Historical and Recent Trends in the Forecasting Literature
Talk for the WARN-D Friday Science Meeting
February 16, 2024
Introduction
Why talk about this topic?
the secret reason:
Ok but really, why should you care?
“The quiet revolution of numerical weather prediction” (Bauer, Thorpe, and Brunet 2015)
Forecasting Competitions
History
Critical statisticians
We should be looking for the true model!
Maybe you do not know how to model with ARIMA…
I suspect it is more likely to depend on the skill of the analyst … these authors are more at home with simple procedures than with Box-Jenkins. (Chatfield)
Makridakis & Hibon (1979)
Our Empirical evidence disagrees
Of course we do
might be useful for Dr. Chatfield to read some of the psychological literature quoted in the main paper, and he can then learn a little more about biases
M-Competitions
| Competition | Year | N° Time Series | Insights/Novelty |
|---|---|---|---|
| M1 | 1982 | 1001 |
|
| M2 | 1993 | 29 |
|
| M3 | 2000 | 3003 |
|
| M4 | 2020 | 100,000 |
|
| M5 | 2021 | 42,000 |
|
| M6 | 2022 | 100 |
|
Utility of Competitions
Advantages
- Empirical evidence
- Benchmarking
- Methodological development and cumulative science (Fildes and Ord 2004)
Issues
- arbitrary conditions
- vibe-based analysis of results (Koning et al. 2005)
- Questionable generalizability (Fildes and Ord 2004)
Model Selection, Uncertainty & Combination
Model selection
The beauty of simple models:
Simple models often outperform more complex ones, even on larger data sets (M1, M3)
Simple models are important benchmarks, even in more complex settings
Selecting a single model:
- ignores uncertainty about this selection (Kaplan 2021)
- tends to perform poorly in forecasting competitions
Model combination
‘Law’ of forecast combination
‘The results have been virtually unanimous: combining multiple forecasts leads to increased forecast accuracy. In many cases one can make dramatic performance improvements by simply averaging the forecasts.’ (Clemen 1989)
Important:
- Which models to choose? Go diverse (Wang et al. 2023)
- How to aggregate? Hard to beat the mean (Aastveit et al., n.d.)
Ensemble modeling
Two sources of uncertainty:
Met Office UK & Bauer, Thorpe, and Brunet (2015)
Relevance for psychology
- Model uncertainty often neglected, both in inferential and predictive modeling (Kaplan 2021)
- Theoretically linked to the complex systems literature
- Helps us understand which methods work when
Probabilistic Forecasting
Probabilistic forecasting methods
What is probabilistic forecasting?
- what is even meant here?
- show plot by gneiting in the appendix
- show different forms
Moving towards probabilistic forecasting
Long history of probabilistic weather forecasting
The probability of rain was much smaller than at other times (Dalton, 1793)
- Popularized by Epstein 1969 Stochastic dynamic prediction
Some more stuff on probabilistic forecasting
Relevance for psychology
- Practically linked to decision theory, e.g., for JITAIs
Mixed Models
Extending mixed models
Flexible Mixed Model
From
\[ y = X\beta + Z\upsilon + \epsilon \]
to
\[y = ml_{fixed}(X)+Z\upsilon + \epsilon\]
- not ‘forecasting’ literature, but ML more broadly
- use of random effects in machine learning has gained attention
Relevance for Psychology
- Current combinations of ‘idiographic’ and ‘nomethetic’ forecasts: often ad-hoc
- Improving on what we already have
Summary
Issues
Methodology
- Reproducibility Issues (Boylan et al. 2015)
- ‘Winner takes it all’? (see Strobl and Leisch (2022)
Transfer to Psychology
- Data structure often not comparable (Makridakis, Hyndman, and Petropoulos 2020)
- Much of it is known, but not used
Takeaways
- Forecasting competitions can lead to methodological improvement
- Model uncertainty and model combination are integral parts of forecasting
- Probabilistic forecasting is important for decision making
- Combining predictions is challenging, but lots of cool stuff in the making
